Parallel prediction of multiple image aspects

ABSTRACT

Example embodiments that analyze images to characterize aspects of the images rely on a same neural network to characterize multiple aspects in parallel. Because additional neural networks are not required for additional aspects, such an approach scales with increased aspects.

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to image search.

BACKGROUND

Present techniques that analyze images to characterize aspects of theimages rely on manual techniques that do not scale. Automated techniquesuse neural networks to characterize aspects of the images. However,because each aspect uses its own neural network, automated techniquesalso do not scale as the number of aspects increases.

BRIEF DESCRIPTION OF THE DRAWINGS

Various ones of the appended drawings merely illustrate exampleembodiments of the present disclosure and cannot be considered aslimiting its scope.

FIG. 1 is a block diagram lustrating a networked system, according tosome example embodiments.

FIG. 2 is a diagram illustrating the operation of the intelligentassistant, according to some example embodiments.

FIG. 3 illustrates the features of the artificial intelligence (AI)framework, according to some example embodiments.

FIG. 4 is a diagram illustrating a service architecture according tosome example embodiments.

FIG. 5 is a block diagram for implement the AI framework, according tosome example embodiments.

FIG. 6 depicts a diagram of a category hierarchy tree that arranges eachpublications of a publication corpus into a hierarchy in accordance withsome example embodiments.

FIG. 7 is an example process flow of training a machine learned model.

FIGS. 8-11 are example process flows of providing, in parallel, multipleaspect probabilities of an input image.

FIG. 12 is a diagram illustrating heat maps of location data oflocations in the input image that have an aspect value.

FIG. 13 is a block diagram illustrating an example of a softwarearchitecture that may be installed on a machine, according to someexample embodiments.

The headings provided herein are merely for convenience and do notnecessarily affect the scope or meaning of the terms used.

DETAILED DESCRIPTION

The description that follows describes systems, methods, techniques,instruction sequences, and computing machine program products thatillustrate example embodiments of the present subject matter. In thefollowing description, for purposes of explanation, numerous specificdetails are set forth in order to provide an understanding of variousembodiments of the present subject matter. It will be evident, however,to those skilled in the art, that embodiments of the present subjectmatter may be practiced without some or other of these specific details.In general, well-known instruction instances, protocols, structures, andtechniques have not been shown in detail. Examples merely typifypossible variations. Unless explicitly stated otherwise, structures(e.g., structural components, such as modules) are optional and may becombined or subdivided, and operations (e.g., in a procedure, algorithm,or other function) may vary in sequence or be combined or subdivided.

Example embodiments that analyze images to characterize aspects of theimages rely on a same neural network to characterize multiple aspects inparallel. Because additional neural networks are not required foradditional aspects, such an approach scales with increased aspects.

FIG. 1 is a block diagram illustrating a networked system, according tosome example embodiments. With reference to FIG. 1, an exampleembodiment of a high-level client-server-based network architecture 100is shown. A networked system 102, in the example forms of anetwork-based marketplace or payment system, provides server-sidefunctionality via a network 104 (e.g., the Internet or wide area network(WAN)) to one or more client devices 110. FIG. 1 illustrates, forexample, a web client 112 (e.g., a browser, such as the InternetExplorer® browser developed by Microsoft® Corporation of Redmond,Wash.), an application 114, and a programmatic client 116 executing onclient device 110.

The client device 110 may comprise, but are not limited to, a mobilephone, desktop computer, laptop, portable digital assistants (PDAs),smart phones, tablets, ultra books, netbooks, laptops, multi-processorsystems, microprocessor-based or programmable consumer electronics, gameconsoles, set-top boxes, or any other communication device that a usermay utilize to access the networked system 102. In some embodiments, theclient device 110 may comprise a display module (not shown) to displayinformation (e.g., in the form of user interfaces). In furtherembodiments, the client device 110 may comprise one or more of a touchscreens, accelerometers, gyroscopes, cameras, microphones, globalpositioning system (GPS) devices, and so forth. The client device 110may be a device of a user that is used to perform a transactioninvolving digital items within the networked system 102. In oneembodiment, the networked system 102 is a network-based marketplace thatresponds to requests for product listings, publishes publicationscomprising item listings of products available on the network-basedmarketplace, and manages payments for these marketplace transactions.One or more users 106 may be a person, a machine, or other means ofinteracting with client device 110. In embodiments, the user 106 is notpart of the network architecture 100, but may interact with the networkarchitecture 100 via client device 110 or another means. For example,one or more portions of network 104 may be an ad hoc network, anintranet, an extranet, a virtual private network (VPN), a local areanetwork (LAN), a wireless LAN (WLAN), a wide area network (WAN), awireless WAN (WWAN), a metropolitan area network (MAN), a portion of theInternet, a portion of the Public Switched Telephone Network (PSTN), acellular telephone network, a wireless network, a WiFi network, a WiMaxnetwork, another type of network, or a combination of two or more suchnetworks.

Each of the client device 110 may include one or more applications (alsoreferred to as “apps”) such as, but not limited to, a web browser,messaging application, electronic mail (email) application, ane-commerce site application (also referred to as a marketplaceapplication), and the like. In some embodiments, if the e-commerce siteapplication is included in a given one of the client device 110, thenthis application is configured to locally provide the user interface andat least some of the functionalities with the application configured tocommunicate with the networked system 102, on an as needed basis, fordata or processing capabilities not locally available (e.g., access to adatabase of items available for sale, to authenticate a user, to verifya method of payment, etc.). Conversely if the e-commerce siteapplication is not included in the client device 110, the client device110 may use its web browser to access the e-commerce site (or a variantthereof) hosted on the networked system 102.

One or more users 106 may be a person, a machine, or other means ofinteracting with the client device 110. In example embodiments, the user106 is not part of the network architecture 100, but may interact withthe network architecture 100 via the client device 110 or other means.For instance, the user provides input (e.g., touch screen input oralphanumeric input) to the client device 110 and the input iscommunicated to the networked system 102 via the network 104. In thisinstance, the networked system 102, in response to receiving the inputfrom the user, communicates information to the client device 110 via thenetwork 104 to be presented to the user. In this way, the user caninteract with the networked system 102 using the client device 110.

An application program interface (API) server 216 and a web server 218are coupled to, and provide programmatic and web interfaces respectivelyto, one or more application servers 140. The application server 140 hostthe intelligent personal assistant system 142, which includes theartificial intelligence framework 144, each of which may comprise one ormore modules or applications and each of which may be embodied ashardware, software, firmware, or any combination thereof.

The application server 140 is, in turn, shown to be coupled to one ormore database servers 226 that facilitate access to one or moreinformation storage repositories or databases 226. In an exampleembodiment, the databases 226 are storage devices that store informationto be posted (e.g., publications or listings) to the publication system242. The databases 226 may also store digital item information inaccordance with example embodiments.

Additionally, a third-party application 132, executing on third-partyservers 130, is shown as having programmatic access to the networkedsystem 102 via the programmatic interface provided by the API server216. For example, the third-party application 132, utilizing informationretrieved from the networked system 102, supports one or more featuresor functions on a website hosted by the third party. The third-partywebsite, for example, provides one or more promotional, marketplace, orpayment functions that are supported by the relevant applications of thenetworked system 102.

Further, while the client-server-based network architecture 100 shown inFIG. 1 employs a client-server architecture, the present inventivesubject matter is of course not limited to such an architecture, andcould equally well find application in a distributed, or peer-to-peer,architecture system, for example. The various publication system 142,payment system 144, and personalization system 150 could also beimplemented as standalone software programs, which do not necessarilyhave networking capabilities.

The web client 212 may access the intelligent personal assistant system142 via the web interface supported by the web server 218. Similarly,the programmatic client 116 accesses the various services and functionsprovided by the intelligent personal assistant system 142 via theprogrammatic interface provided by the API server 216.

Additionally, a third-party application(s) 208, executing on athird-party server(s) 130, is shown as having programmatic access to thenetworked system 102 via the programmatic interface provided by the APIserver 114. example, the third-party application 208, utilizinginformation retrieved from the networked system 102, may support one ormore features or functions on a website hosted by the third party. Thethird-party website may, for example, provide one or more promotional,marketplace, or payment functions that are supported by the relevantapplications of the networked system 102.

FIG. 2 is a diagram illustrating the operation of the intelligentassistant, according to some example embodiments. Today's onlineshopping is impersonal, unidirectional, and not conversational. Buyerscannot speak in plain language to convey their wishes, making itdifficult to convey intent. Shopping on a commerce site is usually moredifficult than speaking with a salesperson or a friend about a product,so oftentimes buyers have trouble finding the products they want.

Embodiments present a personal shopping assistant, also referred to asan intelligent assistant, that supports a two-way communication with theshopper to build context and understand the intent of the shopper,enabling delivery of better, personalized shopping results. Theintelligent assistant has a natural, human-like dialog, that helps abuyer with ease, increasing the likelihood that the buyer will reuse theintelligent assistant for future purchases.

The artificial intelligence framework 144 understands the user and theavailable inventory to respond to natural-language queries and has theability to deliver a incremental improvements in anticipating andunderstanding the customer and their needs.

The artificial intelligence framework (AIF) 144 includes a dialoguemanager 504, natural language understanding (NLU) 206, computer vision208, speech recognition 210, search 218, and orchestrator 220. The AIF144 is able to receive different kinds of inputs, such as text input212, image input 214 and voice input 216, to generate relevant results222. As used herein, the AIF 144 includes a plurality of services (e.g.,NLU 206, computer vision 208) that are implemented by correspondingservers, and the terms service or server may be utilized to identify theservice and the corresponding service.

The natural language understanding (NLU) 206 unit processes naturallanguage text input 212, both formal and informal language, detects theintent of the text, and extracts useful information, such as objects ofinterest and their attributes. The natural language user input can thusbe transformed into a structured query using rich information fromadditional knowledge to enrich the query even further. This informationis then passed on to the dialog manager 504 through the orchestrator 220for further actions with the user or with the other components in theoverall system. The structured and enriched query is also consumed bysearch 218 for improved matching. The text input may be a query for aproduct, a refinement to a previous query, or other information to anobject of relevance (e.g., shoe size).

The computer vision 208 takes image as an input and performs imagerecognition to identify the characteristics of the image (e.g., item theuser wants to ship), which are then transferred to the NLU 206 forprocessing. The speech recognition 210 takes speech 216 as an input andperforms language recognition to convert speech to text, which is thentransferred to the NLU for processing.

The NLU 206 determines the object, the aspects associated with theobject, how to create the search interface input, and how to generatethe response. For example, the AI 144 may ask questions to the user toclarify what the user is looking for. This means that the AIF 144 notonly generates results, but also may create a series of interactiveoperations to get to the optimal, or close to optimal, results 222.

For example, in response to the query, “Can you find me a pair of rednike shoes?” the AIF 144 may generate the following parameters:<intent:shopping, statement-type:question, dominant-object:shoes,target:self, color:red, brand:nike>. To the query, “I am looking for apair of sunglasses for my wife,” the NLU may generate <intent: shopping,statement-type: statement, dominant-object: sunglasses, target:wife,target-gender:female>.

The dialogue manager 504 is the module that analyzes the query of a userto extract meaning, and determines if there is a question that needs tobe asked in order to refine the query, before sending the query tosearch 218. The dialogue manager 504 uses the current communication incontext of the previous communication between the user and theartificial intelligence framework 144. The questions are automaticallygenerated dependent on the combination of the accumulated knowledge(e.g., provided by a knowledge graph) and what search can extract out ofthe inventory. The dialogue manager's job is to create a response forthe user. For example, if the user says, “hello,” the dialogue manager504 generates a response, “Hi, my name is bot.”

The orchestrator 220 coordinates the interactions between the otherservices within the artificial intelligence framework 144. More detailsare provided below about the interactions of the orchestrator 220 withother services with reference to FIG. 5.

FIG. 3 illustrates the features of the artificial intelligence (AI)framework 144, according to some example embodiments. The AIF 144 isable to interact with several input channels 304, such as nativecommerce applications, chat applications, social networks, browsers,etc. In addition, the AIF 144 understands the intent 306 expressed bythe user. For example, the intent may include a user looking for a gooddeal, or a user looking for a gift, or a user on a mission to buy aspecific product, a user looking for suggestions, etc.

Further, the AIF 144 performs proactive data extraction 310 frommultiple sources, such as social networks, email, calendar, news, markettrends, etc. The AIF 144 knows about user details 312, such as userpreferences, desired price ranges, sizes, affinities, etc. The AIF 144facilitates a plurality of services within the service network, such asproduct search, personalization, recommendations, checkout features,etc. The output 308 may include recommendations, results, etc.

The AIF 144 is an intelligent and friendly system that understands theuser's intent (e.g., targeted search, compare, shop, browse), mandatoryparameters (e.g., product, product category, item), optional parameters(e.g., aspects of the item, color, size, occasion), as well as implicitinformation (e.g., geo location, personal preferences, age, gender). TheAIF 144 responds with a well designed response in plain language.

For example, the AIF 144 may process inputs queries, such as: “Hey! Canyou help me find a pair of light pink shoes for my girlfriend please?With heels. Up to $200. Thanks;” “I recently searched for a men'sleather jacket with a classic James Dean look. Think almost HarrisonFord's in the new Star Wars movie. However, I'm looking for quality in aprice range of $200-300. Might not be possible, but I wanted to see!”;or “I'm looking for a black Northface Thermoball jacket.”

Instead of a hardcoded system, the AIF 144 provides a configurable,flexible interface with machine learning capabilities for ongoingimprovement. The AIF 144 supports a commerce system that provides value(connecting the user to the things that the user wants), intelligence(knowing and learning from the user and the user behavior to recommendthe right items), convenience (offering a plurality of user interfaces),easy of-use, and efficiency (saves the user time and money).

FIG. 4 is a diagram illustrating a service architecture 400 according tosome embodiments. The service architecture 400 presents various views ofthe service architecture in order to describe how the servicearchitecture may be deployed on various data centers or cloud services.The architecture 400 represents a suitable environment forimplementation of the embodiments described herein.

The service architecture 402 represents how a cloud architecturetypically appears to a user, developer and so forth. The architecture isgenerally an abstracted representation of the actual underlyingarchitecture implementation, represented in the other views of FIG. 1.For example, the service architecture 402 comprises a plurality oflayers, that represent different functionality and/or servicesassociated with the service architecture 402.

The experience service layer 404 represents a logical grouping ofservices and features from the end customer's point of view, builtacross different client platforms, such as applications running on aplatform (mobile phone, desktop, etc.), web based presentation (mobileweb, desktop web browser, etc.), and so forth. It includes renderinguser interfaces and providing information to the client platform so thatappropriate user interfaces can be rendered, capturing client input, andso forth. In the context of a marketplace, examples of services thatwould reside in this layer are home page (e.g., home view), view itemlisting, search/view search results, shopping cart, buying userinterface and related services, selling user interface and relatedservices, after sale experiences (posting a transaction, feedback,etc.), and so forth. In the context of other systems, the experienceservice layer 404 would incorporate those end user services andexperiences that are embodied by the system.

The API layer 406 contains APIs which allow interaction with businessprocess and core layers. This allows third party development against theservice architecture 402 and allows third parties to develop additionalservices on top of the service architecture 402.

The business process service layer 408 is where the business logicresides for the services provided. In the context of a marketplace thisis where services such as user registration, user sign in, listingcreation and publication, add to shopping cart, place an offer,checkout, send invoice, print labels, ship item, return item, and soforth would be implemented. The business process service layer 408 alsoorchestrates between various business logic and data entities and thusrepresents a composition of shared services. The business processes inthis layer can also support multi-tenancy in order to increasecompatibility with some cloud service architectures.

The data entity service layer 410 enforces isolation around direct dataaccess and contains the services upon which higher level layers depend.Thus, in the marketplace context this layer can comprise underlyingservices like order management, financial institution management, useraccount services, and so forth. The services in this layer typicallysupport multi-tenancy.

The infrastructure service layer 412 comprises those services that arenot specific to the type of service architecture being implemented.Thus, in the context of a marketplace, the services in this layer areservices that are not specific or unique to a marketplace. Thus,functions like cryptographic functions, key management, CAPTCHA,authentication and authorization, configuration management, logging,tracking, documentation and management, and so forth reside in thislayer.

Embodiments of the present disclosure will typically be implemented inone or more of these layers. In particular, the AIF 144, as well as theorchestrator 220 and the other services of the AIF 144.

The data center 414 is a representation of the various resource pools416 along with their constituent scale units. This data centerrepresentation illustrates the scaling and elasticity that comes withimplementing the service architecture 402 in a cloud computing model.The resource pool 416 is comprised of server (or compute) scale units420, network scale units 418 and storage scale units 422. A scale unitis a server, network and/or storage unit that is the smallest unitcapable of deployment within the data center. The scale units allow formore capacity to be deployed or removed as the need increases ordecreases.

The network scale unit 418 contains one or more networks (such asnetwork interface units, etc.) that can be deployed. The networks caninclude, for example virtual LANs. The compute scale unit 420 typicallycomprise a unit (server, etc.) that contains a plurality processingunits, such as processors. The storage scale unit 422 contains one ormore storage devices such as disks, storage attached networks (SAN),network attached storage (NAS) devices, and so forth. These arecollectively illustrated as SANs in the description below. Each SAN maycomprise one or more volumes, disks, and so forth.

The remaining view of FIG. 1 illustrates another example of a servicearchitecture 400. This view is more hardware focused and illustrates theresources underlying the more logical architecture in the other views ofFIG. 1. A cloud computing architecture typically has a plurality ofservers or other systems 424, 426. These servers comprise a plurality ofreal and/or virtual servers. Thus the server 424 comprises server 1along with virtual servers 1A, 1B, 1C and so forth.

The servers are connected to and/or interconnected by one or morenetworks such as network A 428 and/or network B 430. The servers arealso connected to a plurality of storage devices, such as SAN 1 (436),SAN 2 (438) and so forth. SANs are typically connected to the serversthrough a network such as SAN access A 432 and/or SAN access B 434.

The compute scale units 420 are typically some aspect of servers 424and/or 426, like processors and other hardware associated therewith. Thenetwork scale units 418 typically include, or at least utilize theillustrated networks A (428) and B (432). The storage scale unitstypically include some aspect of SAN 1 (436) and/or SAN 2 (438). Thus,the logical service architecture 402 can be mapped to the physicalarchitecture.

Services and other implementation of the embodiments described hereinwill run on the servers or virtual servers and utilize the varioushardware resources to implement the disclosed embodiments.

FIG. 5 is a block diagram for implement the AIF 144, according to someexample embodiments. Specifically, the intelligent personal assistantsystem 106 of FIG. 2 is shown to include a front end component 502 (FE)by which the intelligent personal assistant system 106 communicates(e.g., over the network 104) with other systems within the networkarchitecture 100. The front end component 502 can communicate with thefabric of existing messaging systems. As used herein, the term messagingfabric refers to a collection of APIs and services that can power thirdparty platforms such as Facebook messenger, Microsoft Cortana, andothers “bots.” In one example, a messaging fabric can support an onlinecommerce ecosystem that allows users to interact with commercial intent.Output of the front end component 502 can be rendered in a display of aclient device, such as the client device 110 in FIG. 1 as part of aninterface with the intelligent personal assistant.

The front end component 502 of the intelligent personal assistant system106 is coupled to a back end component 504 for the front end (BFF) thatoperates to link the front end component 502 with the AIF 144. Theartificial intelligence framework 144 includes several componentsdiscussed below.

In one example embodiment, an orchestrator 220 orchestratescommunication of components inside and outside the artificialintelligence framework 144. Input modalities for the AI orchestrator 206are derived from a computer vision component 208, a speech recognitioncomponent 210, and a text normalization component which may form part ofthe speech recognition component 210. The computer vision component 208may identify objects and attributes from visual input (e.g., photo). Thespeech recognition component 210 converts audio signals (e.g., spokenutterances) into text. The text normalization component operates to makeinput normalization, such as language normalization by renderingemoticons into text, for example. Other normalization is possible suchas orthographic normalization, foreign language normalization,conversational text normalization, and so forth.

The artificial intelligence framework 144 further includes a naturallanguage understanding (NLU) component 206 that operates to parse andextract user intent and intent parameters (for example mandatory oroptional parameters). The NLU component 206 is shown to includesub-components such as a spelling corrector (speller), a parser, a namedentity recognition (NER) sub-component, a knowledge graph, and a wordsense detector (WSD).

The artificial intelligence framework 144 further includes a dialogmanager 204 that operates to understand a “completeness of specificity”(for example of an input, such as a search query or utterance) anddecide on a next action type and a parameter (e.g., “search” or “requestfurther information from user”). In one example, the dialog manager 204operates in association with a context manager 518 and a naturallanguage generation (NLG) component 512. The context manager 518 managesthe context and communication of a user with respect to online personalassistant (or “bot”) and the assistant's associated artificialintelligence. The context manager 518 comprises two parts: long termhistory and short term memory. Data entries into one or both of theseparts can include the relevant intent and all parameters and all relatedresults of a given input, bot interaction, or turn of communication, forexample. The NLG component 512 operates to compose a natural languageutterance out of a AI message to present to a user interacting with theintelligent bot.

A search component 218 is also included within the artificialintelligence framework 144. As shown, the search component 218 has afront-end and a back-end unit. The back-end unit operates to manage itemand product inventory and provide functions of searching against theinventory, optimizing towards a specific tuple of intent and intentparameters. An identity service 522 component, that may or may not formpart of artificial intelligence framework 144, operates to manage userprofiles, for example explicit information in the form of userattributes (e.g., “name,” “age,” “gender,” “geolocation”), but alsoimplicit information in forms such as “information distillates” such as“user interest,” or “similar persona,” and so forth. The identityservice 522 includes a set of policies, APIs, and services thatelegantly centralizes all user information, enabling the AIF 144 to haveinsights into the users' wishes. Further, the identity service 522protects the commerce system and its users from fraud or malicious useof private information.

The functionalities of the artificial intelligence framework 144 can beset into multiple parts, for example decision-making and context parts.In one example, the decision-making part includes operations by theorchestrator 220, the NLU component 206 and its subcomponents, thedialog manager 204, the NLG component 512, the computer vision component208 and speech recognition component 210. The context part of the AIfunctionality relates to the parameters (implicit and explicit) around auser and the communicated intent (for example, towards a giveninventory, or otherwise). In order to measure and improve AI qualityover time, in some example embodiments, the artificial intelligenceframework 144 is trained using sample queries (e.g., a development set)and tested on a different set of queries (e.g., an evaluation set), bothsets to be developed by human curation or from use data. Also, theartificial intelligence framework 144 is to be trained on transactionand interaction flows defined by experienced curation specialists, orhuman override 524. The flows and the logic encoded within the variouscomponents of the artificial intelligence framework 144 define whatfollow-up utterance or presentation (e.g., question, result set) is madeby the intelligent assistant based on an identified user intent.

The intelligent personal assistant system 106 seeks to understand auser's intent (e.g., targeted search, compare, shop, browse, and soforth), mandatory parameters (e.g., product, product category, item, andso forth), and optional parameters (e.g., explicit information, e.g.,aspects of item/product, occasion, and so forth), as well as implicitinformation (e.g., geolocation, personal preferences, age and gender,and so forth) and respond to the user with a content-rich andintelligent response. Explicit input modalities can include text,speech, and visual input and can be enriched with implicit knowledge ofuser (e.g., geolocation, gender, birthplace, previous browse history,and so forth). Output modalities can include text (such as speech, ornatural language sentences, or product-relevant information, and imageson the screen of a smart device e.g., client device 110. Inputmodalities thus refer to the different ways users can communicate withthe bot. Input modalities can also include keyboard or mouse navigation,touch-sensitive gestures, and so forth.

In relation to a modality for the computer vision component 208, aphotograph can often represent what a user is looking for better thantext. Also, the computer vision component 208 may be used to formshipping parameters based on the image of the item to be shipped. Theuser may not know what an item is called, or it may be hard or evenimpossible to use text for fine detailed information that an expert mayknow, for example a complicated pattern in apparel or a certain style infurniture. Moreover, it is inconvenient to type complex text queries onmobile phones and long text queries typically have poor recall. Keyfunctionalities of the computer vision component 208 include objectlocalization, object recognition, optical character recognition (OCR)and matching against inventory based on visual cues from an image orvideo. A bot enabled with computer vision is advantageous when runningon a mobile device which has a built-in camera. Powerful deep neuralnetworks can be used to enable computer vision applications.

With reference to the speech recognition component 210, a featureextraction component operates to convert raw audio waveform tosome-dimensional vector of numbers that represents the sound. Thiscomponent uses deep learning to project the raw signal into ahigh-dimensional semantic space. An acoustic model component operates tohost a statistical model of speech units, such as phonemes andallophones. These can include Gaussian Mixture Models (GMM) although theuse of Deep Neural Networks is possible. A language model component usesstatistical models of grammar to define how words are put together in asentence. Such models can include n-gram-based models or Deep NeuralNetworks built on top of word embeddings. A speech-to-text (STT) decodercomponent converts a speech utterance into a sequence of words typicallyleveraging features derived from a raw signal using the featureextraction component, the acoustic model component, and the languagemodel component in a Hidden Markov Model (HMM) framework to derive wordsequences from feature sequences. In one example, a speech-to-textservice in the cloud has these components deployed in a cloud frameworkwith an API that allows audio samples to be posted for speech utterancesand to retrieve the corresponding word sequence. Control parameters areavailable to customize or influence the speech-to-text process.

Machine-learning algorithms may be used for matching, relevance, andfinal re-ranking by the AIF 144 services. Machine learning is a field ofstudy that gives computers the ability to learn without being explicitlyprogrammed. Machine learning explores the study and construction ofalgorithms that can learn from and make predictions on data. Suchmachine-learning algorithms operate by building a model from exampleinputs in order to make data-driven predictions or decisions expressedas outputs. Machine-learning algorithms may also be used to teach how toimplement a process.

Deep learning models, deep neural network (DNN), recurrent neuralnetwork (RNN), convolutional neural network (CNN), and long short-termCNN, as well as other ML models and IR models may be used. For example,search 218 may use n-gram, entity, and semantic vector-based query toproduct matching. Deep-learned semantic vectors give the ability tomatch products to non-text inputs directly. Multi-leveled relevancefiltration may use BM25, predicted query leaf category+product leafcategory, semantic vector similarity between query and product, andother models, to pick the top candidate products for the finalre-ranking algorithm.

Predicted click-through-rate and conversion rate, as well as GMV,constitutes the final re-ranking formula to tweak functionality towardsspecific business goals, more shopping engagement, more productspurchased, or more GMV. Both the click prediction and conversionprediction models take in query, user, seller and product as inputsignals. User profiles are enriched by learning from onboarding,sideboarding, and user behaviors to enhance the precision of the modelsused by each of the matching, relevance, and ranking stages forindividual users. To increase the velocity of model improvement, offlineevaluation pipeline is used before online A/B testing.

In one example of an artificial intelligence framework 144, twoadditional parts for the speech recognition component 210 are provided,a speaker adaptation component and an LM adaptation component. Thespeaker adaptation component allows clients of an STT system (e.g.,speech recognition component 210) to customize the feature extractioncomponent and the acoustic model component for each speaker. This can beimportant because most speech-to-text systems are trained on data from arepresentative set of speakers from a target region and typically theaccuracy of the system depends heavily on how well the target speakermatches the speakers in the training pool. The speaker adaptationcomponent allows the speech recognition component 210 (and consequentlythe artificial intelligence framework 144) to be robust to speakervariations by continuously learning the idiosyncrasies of a user'sintonation, pronunciation, accent, and other speech factors and applythese to the speech-dependent components, e.g., the feature extractioncomponent, and the acoustic model component. While this approachutilizes a non-significant-sized voice profile to be created andpersisted for each speaker, the potential benefits of accuracy generallyfar outweigh the storage drawbacks.

The language model (LM) adaptation component operates to customize thelanguage model component and the speech-to-text vocabulary with newwords and representative sentences from a target domain, for example,inventory categories or user personas. This capability allows theartificial intelligence framework 144 to be scalable as new categoriesand personas are supported.

The AIF's goal is to provide a scalable and expandable framework for AI,one in which new activities, also referred to herein as missions, can beaccomplished dynamically using the services that perform specificnatural-language processing functions. Adding a new service does notrequire to redesign the complete system. Instead, the services areprepared (e.g., using machine-learning algorithms) if necessary, and theorchestrator is configured with a new sequence related to the newactivity. More details regarding the configuration of sequences areprovided below with reference to other figures and associated text.

Embodiments presented herein provide for dynamic configuration of theorchestrator 220 to learn new intents and how to respond to the newintents. In some example embodiments, the orchestrator 220 “learns” newskills by receiving a configuration for a new sequence associated withthe new activity. The sequence specification includes a sequence ofinteractions between the orchestrator 220 and a set of one or moreservice servers from the AIF 144. In some example embodiments, eachinteraction of the sequence includes (at least): identification for aservice server, a call parameter definition to be passed with a call tothe identified service server; and a response parameter definition to bereturned by the identified service server.

In some example embodiments, the services within the Alf 144, except forthe orchestrator 220, are not aware of each other, e.g., they do notinteract directly with each other. The orchestrator 220 manages all theinteractions with the other servers. Having the central coordinatingresource simplifies the implementation of the other services, which neednot be aware of the interfaces (e.g., APIs) provided by the otherservices. Of course, there can be some cases where a direct interfacemay be supported between pairs of services.

FIG. 6 depicts a diagram of a category hierarchy tree that arranges eachpublications of a publication corpus into a hierarchy in accordance withsome example embodiments. In some example embodiments, the publicationcategories are then organized into a hierarchy (e.g., a map or tree),such that more general categories include more specific categories. Eachnode in the tree or map is a publication category that has a parentcategory (e.g., a more general category with which the publicationcategory is associated) and potentially one or more child categories(e.g., narrow or more specific categories associated with thepublication category.). Each publication category is associated with aparticular static webpage.

In accordance with some example embodiments, a plurality of publicationis grouped together into publication categories. In this example, eachcategory is labeled with a letter (e.g., category A-category AJ). Inaddition, every publication category is organized as part of a hierarchyof categories.

In this example, category A is a general product category that all otherpublication categories descend from. Publications in category A are thendivided in to at least two different publication categories, category Band category C. It should be noted that each parent category (e.g., inthis case category A is a parent category to both Category B andCategory C) may include a large number of child categories (e.g.,subcategories).

In this example, publication categories B and C both have subcategories(or child categories). For example, if Category A is clothingpublications, Category B can be Men's clothes publications and CategoryC is Women's clothes publications. Subcategories for Category B includecategory D, category E, and category F. Each of subcategories D, E, andF have a different number of subcategories, depending on the specificdetails of the publications covered by each subcategory.

For example, if category D is active wear publications, category E isformal wear publications, and category F is outdoor wear publications,each subcategory includes different numbers and types of subcategories.For example, category D (active wear publications in this example)includes subcategories I and J. Subcategory I includes Active Footwearpublications (for this example) and Subcategory J includes t-shirtpublications. As a result of the differences between these twosubcategories, subcategory I includes four additional subcategories(subcategories K-N) to represent different types of active footwearpublications (e.g., running shoe publications, basketball shoepublications, climbing shoe publications, and tennis shoe publications).In contrast, subcategory J (which, in this example, is for t-shirtpublications) does not include any subcategories (although in a realproduct database a t-shirt publications category would likely includesubcategories).

Thus, each category has a parent category (except for the uppermostproduct category) which represents a more general category ofpublications and one or more child categories or subcategories (whichare a more specific publications category within the more generalcategory). Thus, category E has two sub-categories, O and P, and eachsubcategory has two child product categories, categories Q and R andcategories S and T, respectively. Similarly, category F has threesub-categories (U, V, and W).

Category C, a product category that has Category A as its parent,includes two additional subcategories (G and H). Category G includes twochildren (X and AF). Category X includes subcategories Y and Z, and Yincludes AA-AE. Category H includes subcategories AG and AH. Category AGincludes categories AI and AJ.

Aspects are characteristics of publications listed in a given category.Aspects are specific to a specific category, Aspects of differentcategories may or may not overlap, or be unique to a specific category.

FIG. 7 is an example process flow of training a machine learned model.At 710, a training image is input to a machine learned model. At 720,the training image is processed with the machine learned model. At 730,multiple training aspect values are output in parallel from the machinelearned model. At 740, the machine learned model is trained by feedingback to the machine learned model whether or not each of the multipletraining aspect values was correct.

In an example embodiment, a machine-learned model is used to embed thedeep latent semantic meaning of a given listing title and project it toa shared semantic vector space. A vector space can be referred to as acollection of objects called vectors. Vectors spaces can becharacterized by their dimension, which specifies the number ofindependent directions in the space. A semantic vector space canrepresent phrases and sentences and can capture semantics for imagesearch and image characterization tasks. In further embodiments, asemantic vector space can represent audio sounds, phrases, or music;video clips; and images and can capture semantics for image search andimage characterization tasks.

In various embodiments, machine learning is used to maximize thesimilarity between the source (X), for example, a listing title, and thetarget (Y), the search query. A machine-learned model may be based ondeep neural networks (DNN) or convolutional neural networks (CNN). TheDNN is an artificial neural network with multiple hidden layers of unitsbetween the input and output layers. The DNN can apply the deep learningarchitecture to recurrent neural networks. The CNN is composed of one ormore convolution layers with fully connected layers (such as thosematching a typical artificial neural network) on top. The CNN also usestied weights and pooling layers. Both the DNN and CNN can be trainedwith a standard backpropagation algorithm.

When a machine-learned model is applied to mapping a specific <source,target> pair, the parameters for machine-learned Source Model andmachine-learned Target Model are optimized so that relevant <source,target> pair has closer vector representation distance. The followingformula can be used to compute the minimum distance.

${SrcMod}^{*},{{TgtMod}^{*} = {{argmin}{\sum\limits_{k\mspace{14mu}{in}\mspace{14mu}{all}\mspace{14mu}{training}\mspace{14mu}{pairs}}{{{SrcVec}^{k} - {T\;{\mathcal{g}}\;{tVec}^{k}}}}}}}$Where,ScrSeq=a source sequence;TgtSeq=a target sequence;SrcMod=source machine-learned model;TgtMod=target machine-learned model;SrcVec=a continuous vector representation for a source sequence (alsoreferred to the semantic vector of the source); andTgtVec=a continuous vector representation for a target sequence (alsoreferred to as semantic vector of the target).

The source machine-learned model encodes the source sequence into acontinuous vector representation. The target machine-learned modelencodes the target sequence into a continuous vector representation. Inan example embodiment, the vectors each have approximately 100dimensions.

In other embodiments, any number of dimensions may be used. In exampleembodiments, the dimensions of the semantic vectors are stored in a KDtree structure. The KD tree structure can be referred to aspace-partitioning data structure for organizing points in a KD space.The KD tree can be used to perform the nearest-neighbor lookup. Thus,given a source point in space, the nearest-neighbor lookup may be usedto identify the closest point to the source point.

FIGS. 8-11 are example process flows of providing, in parallel, multipleaspect probabilities of an input image. In FIG. 8, at 810 an input imageis transmitted from a device operated by a user. The user may besearching for a publication in a publication corpus. The user may beposting a new publication with publication images, and rely on theprocess flow to help provide the category. At 820, an input semanticvector corresponding to the input image is accessed. At this point, theprocess flow splits. At 830, the input semantic vector and publicationimage vectors are converted into binary representations. At 840, closestmatches are identified between the input semantic vector and publicationimage vectors that are representative of multiple aspects. The machinelearned model is used along with XOR operations for speed. A number ofcommon bits from the XOR operation is a measure of similarity. In analternative flow, at 850 closest matches are identified between theinput semantic vector and publication image vectors that arerepresentative of multiple aspects by finding nearest neighbors insemantic vector space. After either of the previous split process flows,at 870 the multiple aspect probabilities are provided, based on themachine learned model.

The process flow of FIG. 9 is generally similar to FIG. 8. At 910, theinput image is missing category metadata. At 970, at least one aspectprobability is provided for the input image that was missing metadata ofthe aspect value. In another embodiment, at least one aspect probabilityis provided for the input image that was not missing metadata, to doublecheck the metadata.

The process flow of FIG. 10 is generally similar to FIG. 9. At 1070, themissing aspect metadata is added to the input image, responsive to anaspect probability exceeding a minimum threshold.

The process flow of FIG. 11 is generally similar to FIG. 10. At 1170,location metadata is added to the input image, such that the locationmetadata has the location in the input image that has the aspect value.

FIG. 12 is a diagram illustrating heat maps of location data oflocations in the input image that have an aspect value. Images 1210,1220, and 1230 are input images having the aspect value of hoodie.Respective heat maps 1212, 1222, and 1232 are heat maps having thelocations in the images that have the aspect value of hoodie. Forexample, the hot regions of the heat maps correspond to regions of theinput image that have parts of a hoodie, such as a hood, sleeves, andpockets.

FIG. 13 is a block diagram illustrating components of a machine 1300,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein.Specifically, FIG. 13 shows a diagrammatic representation of the machine1300 in the example form of a computer system, within which instructions1310 (e.g., software, a program, an application, an applet, an app, orother executable code) for causing the machine 1300 to perform any oneor more of the methodologies discussed herein may be executed. Forexample, the instructions 1310 may cause the machine 1300 to execute theflow diagrams of other Figures. Additionally, or alternatively, theinstructions 1310 may implement the servers associated with the servicesand components of other Figures, and so forth. The instructions 1310transform the general, non-programmed machine 1300 into a particularmachine 1300 programmed to carry out the described and illustratedfunctions in the manner described.

In alternative embodiments, the machine 1300 operates as a standalonedevice or may be coupled (e.g., networked) to other machines. In anetworked deployment, the machine 1300 may operate in the capacity of aserver machine or a client machine in a server-client networkenvironment, or as a peer machine in a peer-to-peer (or distributed)network environment. The machine 1300 may comprise, but not be limitedto, a switch, a controller, a server computer, a client computer, apersonal computer (PC), a tablet computer, a laptop computer, a netbook,a set-top box (STB), a personal digital assistant (PDA), anentertainment media system, a cellular telephone, a smart phone, amobile device, a wearable device (e.g., a smart watch), a smart homedevice (e.g., a smart appliance), other smart devices, a web appliance,a network router, a network switch, a network bridge, or any machinecapable of executing the instructions 1310, sequentially or otherwise,that specify actions to be taken by the machine 1300. Further, whileonly a single machine 1300 is illustrated, the term “machine” shall alsobe taken to include a collection of machines 1300 that individually orjointly execute the instructions 1310 to perform any one or more of themethodologies discussed herein.

The machine 1300 may include processors 1304, memory/storage 1306, andI/O components 1318, which may be configured to communicate with eachother such as via a bus 1302. In an example embodiment, the processors1304 (e.g., a Central Processing Unit (CPU), a Reduced Instruction SetComputing (RISC) processor, a Complex Instruction Set Computing (CISC)processor, a Graphics Processing Unit (GPU), a Digital Signal Processor(DSP), an Application Specific Integrated Circuit (ASIC), aRadio-Frequency Integrated Circuit (RFIC), another processor, or anysuitable combination thereof) may include, for example, a processor 1308and a processor 1312 that may execute the instructions 1310. The term“processor” is intended to include multi-core processors that maycomprise two or more independent processors (sometimes referred to as“cores”) that may execute instructions contemporaneously. Although FIG.13 shows multiple processors 1304, the machine 1300 may include a singleprocessor with a single core, a single processor with multiple cores(e.g., a multi-core processor), multiple processors with a single core,multiple processors with multiples cores, or any combination thereof.

The memory/storage 1306 may include a memory 1314, such as a mainmemory, or other memory storage, and a storage unit 1313, bothaccessible to the processors 1304 such as via the bus 1302. The storageunit 1313 and memory 1314 store the instructions 1310 embodying any oneor more of the methodologies or functions described herein. Theinstructions 1310 may also reside, completely or partially, within thememory 1314, within the storage unit 1313, within at least one of theprocessors 1304 (e.g., within the processor's cache memory), or anysuitable combination thereof, during execution thereof by the machine1300. Accordingly, the memory 1314, the storage unit 1313, and thememory of the processors 1304 are examples of machine-readable media.

As used herein, “machine-readable medium” means a device able to storeinstructions and data temporarily or permanently and may include, but isnot limited to, random-access memory (RAM), read-only memory (ROM),buffer memory, flash memory, optical media, magnetic media, cachememory, other types of storage (e.g., Erasable Programmable Read-OnlyMemory (EEPROM)), and/or any suitable combination thereof. The term“machine-readable medium” should be taken to include a single medium ormultiple media (e.g., a centralized or distributed database, orassociated caches and servers) able to store the instructions 1310. Theterm “machine-readable medium” shall also be taken to include anymedium, or combination of multiple media, that is capable of storinginstructions (e.g., instructions 1310) for execution by a machine (e.g.,machine 1300), such that the instructions, when executed by one or moreprocessors of the machine (e.g., processors 1304), cause the machine toperform any one or more of the methodologies described herein.Accordingly, a “machine-readable medium” refers to a single storageapparatus or device, as well as “cloud-based” storage systems or storagenetworks that include multiple storage apparatus or devices. The term“machine-readable medium” excludes signals per se.

The I/O components 1318 may include a wide variety of components toreceive input, provide output, produce output, transmit information,exchange information, capture measurements, and so on. The specific I/Ocomponents 1318 that are included in a particular machine will depend onthe type of machine. For example, portable machines such as mobilephones will likely include a touch input device or other such inputmechanisms, while a headless server machine will likely not include sucha touch input device. It will be appreciated that the 110 components1318 may include many other components that are not shown in FIG. 13.The I/O components 1318 are grouped according to functionality merelyfor simplifying the following discussion, and the grouping is in no waylimiting. In various example embodiments, the I/O components 1318 mayinclude output components 1326 and input components 1328. The outputcomponents 1326 may include visual components (e.g., a display such as aplasma display panel (PDP), a light emitting diode (LED) display, aliquid crystal display (LCD), a projector, or a cathode ray tube (CRT)),acoustic components (e.g., speakers), haptic components (e.g., avibratory motor, resistance mechanisms), other signal generators, and soforth. The input components 1328 may include alphanumeric inputcomponents (e.g., a keyboard, a touch screen configured to receivealphanumeric input, a photo-optical keyboard, or other alphanumericinput components), point based input components (e.g., a mouse, atouchpad, a trackball, a joystick, a motion sensor, or other pointinginstruments), tactile input components (e.g., a physical button, a touchscreen that provides location and/or force of touches or touch gestures,or other tactile input components), audio input components (e.g., amicrophone), and the like.

In further example embodiments, the I/O components 1318 may includebiometric components 1330, motion components 1334, environmentalcomponents 1336, or position components 1338 among a wide array of othercomponents. For example, the biometric components 1330 may includecomponents to detect expressions (e.g., hand expressions, facialexpressions, vocal expressions, body gestures, or eye tracking), measurebiosignals (e.g., blood pressure, heart rate, body temperature,perspiration, or brain waves), identify a person (e.g., voiceidentification, retinal identification, facial identification,fingerprint identification, or electroencephalogram basedidentification), and the like. The motion components 1334 may includeacceleration sensor components (e.g., accelerometer), gravitation sensorcomponents, rotation sensor components (e.g., gyroscope), and so forth.The environmental components 1336 may include, for example, illuminationsensor components (e.g., photometer), temperature sensor components(e.g., one or more thermometers that detect ambient temperature),humidity sensor components, pressure sensor components (e.g.,barometer), acoustic sensor components (e.g., one or more microphonesthat detect background noise), proximity sensor components (e.g.,infrared sensors that detect nearby objects), gas sensors (e.g., gasdetection sensors to detect concentrations of hazardous gases for safetyor to measure pollutants in the atmosphere), or other components thatmay provide indications, measurements, or signals corresponding to asurrounding physical environment. The position components 1338 mayinclude location sensor components (e.g., a Global Position System (GPS)receiver component), altitude sensor components (e.g., altimeters orbarometers that detect air pressure from which altitude may be derived),orientation sensor components (e.g., magnetometers), and the like.

Communication may be implemented using a wide variety of technologies.The I/O components 1318 may include communication components 1340operable to couple the machine 1300 to a network 1332 or devices 1320via a coupling 1324 and a coupling 1322, respectively. For example, thecommunication components 1340 may include a network interface componentor other suitable device to interface with the network 1332. In furtherexamples, the communication components 1340 may include wiredcommunication components, wireless communication components, cellularcommunication components, Near Field Communication (NFC) components,Bluetooth® components (e.g., Bluetooth® Low Energy), Wi-Fi® components,and other communication components to provide communication via othermodalities. The devices 1320 may be another machine or any of a widevariety of peripheral devices (e.g., a peripheral device coupled via aUSB).

Moreover, the communication components 1340 may detect identifiers orinclude components operable to detect identifiers. For example, thecommunication components 1340 may include Radio Frequency Identification(RFID) tag reader components, NFC smart tag detection components,optical reader components (e.g., an optical sensor to detectone-dimensional bar codes such as Universal Product Code (UPC) bar code,multi-dimensional bar codes such as Quick Response (QR) code, Azteccode, Data Matrix, Dataglyph, MaxiCode, PDF417, Ultra Code, UCC RSS-2Dbar code, and other optical codes), or acoustic detection components(e.g., microphones to identify tagged audio signals). In addition, avariety of information may be derived via the communication components1340, such as location via Internet Protocol (IP) geo-location, locationvia Wi-Fi® signal triangulation, location via detecting an NFC beaconsignal that may indicate a particular location, and so forth.

In various example embodiments, one or more portions of the network 1332may be an ad hoc network, an intranet, an extranet, a virtual privatenetwork (VPN), a local area network (LAN), a wireless LAN (WLAN), a widearea network (WAN), a wireless WAN (WWAN), a metropolitan area network(MAN), the Internet, a portion of the Internet, a portion of the PublicSwitched Telephone Network (PSTN), a plain old telephone service (POTS)network, a cellular telephone network, a wireless network, a Wi-Fi®network, another type of network, or a combination of two or more suchnetworks. For example, the network 1332 or a portion of the network 1332may include a wireless or cellular network and the coupling 1324 may bea Code Division Multiple Access (CDMA) connection, a Global System forMobile communications (GSM) connection, or another type of cellular orwireless coupling. In this example, the coupling 1324 may implement anyof a variety of types of data transfer technology, such as SingleCarrier Radio Transmission Technology (1×RTT), Evolution-Data Optimized(EVDO) technology, General Packet Radio Service (GPRS) technology,Enhanced Data rates for GSM Evolution (EDGE) technology, thirdGeneration Partnership Project (3GPP) including 3G, fourth generationwireless (4G) networks, Universal Mobile Telecommunications System(UMTS), High Speed Packet Access (HSPA), Worldwide Interoperability forMicrowave Access (WiMAX), Long Term Evolution (LTE) standard, othersdefined by various standard-setting organizations, other long rangeprotocols, or other data transfer technology.

The instructions 1310 may be transmitted or received over the network1332 using a transmission medium via a network interface device (e.g., anetwork interface component included in the communication components1340) and utilizing any one of a number of well-known transfer protocols(e.g., hypertext transfer protocol (I-ITTP)). Similarly, theinstructions 1310 may be transmitted or received using a transmissionmedium via the coupling 1322 (e.g., a peer-to-peer coupling) to thedevices 1320. The term “transmission medium” shall be taken to includeany intangible medium that is capable of storing, encoding, or carryingthe instructions 1310 for execution by the machine 1300, and includesdigital or analog communications signals or other intangible media tofacilitate communication of such software.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

The embodiments illustrated herein are described in sufficient detail toenable those skilled in the art to practice the teachings disclosed.Other embodiments may be used and derived therefrom, such thatstructural and logical substitutions and changes may be made withoutdeparting from the scope of this disclosure. The Detailed Description,therefore, is not to be taken in a limiting sense, and the scope ofvarious embodiments is defined only by the appended claims, along withthe full range of equivalents to which such claims are entitled.

As used herein, the term “or” may be construed in either an inclusive orexclusive sense. Moreover, plural instances may be provided forresources, operations, or structures described herein as a singleinstance. Additionally, boundaries between various resources,operations, modules, engines, and data stores are somewhat arbitrary,and particular operations are illustrated in a context of specificillustrative configurations. Other allocations of functionality areenvisioned and may fall within a scope of various embodiments of thepresent disclosure. In general, structures and functionality presentedas separate resources in the example configurations may be implementedas a combined structure or resource. Similarly, structures andfunctionality presented as a single resource may be implemented asseparate resources. These and other variations, modifications,additions, and improvements fall within a scope of embodiments of thepresent disclosure as represented by the appended claims. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. A method comprising: receiving, by anetwork-based marketplace, a search query for an item listing of thenetwork-based marketplace, the search query including an input image;determining that the input image is missing metadata of one or moreaspect values characterizing the input image, the one or more aspectvalues comprising a category of an item depicted in the input imageand/or a color of the item depicted in the input image; in response tothe determining that the input image is missing metadata, generating aninput semantic vector representing the input image: generating a set ofranked search results in response to the search query, the searchresults comprising a plurality of images; providing the input semanticvector as input to a machine learning system of a publication corpus;identifying, by the machine learning system, closest matches between theinput semantic vector representing the input image and image vectors ofpublications images of the publication corpus, wherein the closestmatches are identified based on an exclusive-or operation providing ameasure of similarity between the input semantic vector and imagevectors of publications images; responsive to said identifying,receiving, as a plurality of parallel outputs from the machine learningsystem, a plurality of aspect probabilities indicating the one or moreaspect values that characterize the input image, the plurality of aspectprobabilities identifying probabilities that the input image has acorresponding aspect value of the one or more aspect values; responsiveto the at least one of the plurality of aspect probabilities of theinput image exceeding a minimum threshold, adding metadata to the inputimage, the metadata including a corresponding aspect value of theplurality of aspect values having the at least one of the plurality ofaspect probabilities; and reranking the search results based on theaspect values indicated by the outputs of the machine learning systemthat were missing from the metadata.
 2. The method of claim 1, wherein,despite the input image having no metadata characterizing any of theplurality of aspect values in the image, at least one of the pluralityof aspect probabilities in the parallel outputs is directed to the atleast one of the plurality of aspect values.
 3. The method of claim 1,further comprising: adding a publication to the publication corpus, thepublication including an image of the plurality of image's respectivemetadata.
 4. The method of claim 1, wherein the parallel outputs includeoutput location data with a location within a respective image that hasan aspect value of the plurality of aspect values.
 5. The method ofclaim 1, further comprising determining a product category for the itemlisting in the network based marketplace based on the indicated one ormore aspect values.
 6. A computer comprising: a storage device storinginstructions; and one or more hardware processors configured by theinstructions to perform operations comprising: receiving, by anetwork-based marketplace, a search query for an item listing of thenetwork-based marketplace, the search query including an input image,the one or more aspect values comprising a category of an item depictedin the input image and/or a color of the item depicted in the inputimage; determining that the input image is missing metadata of one ormore aspect values characterizing the input image; in response to thedetermining that the input image is missing metadata, generating aninput semantic vector representing the input image, generating a set ofranked search results in response to the search query, the searchresults comprising a plurality of images; providing the input semanticvector as input to a machine learning system of a publication corpus;identifying, by the machine learning system, closest matches between theinput semantic vector representing the input image and image vectors ofpublications images of the publication corpus, wherein the closestmatches are identified based on an exclusive-or operation providing ameasure of similarity between the input semantic vector and imagevectors of publications images; responsive to said identifying,receiving, as a plurality of parallel outputs from the machine learningsystem, a plurality of aspect probabilities indicating the one or moreaspect values that characterize the input image, the plurality of aspectprobabilities identifying probabilities that the input image has acorresponding aspect value of the one or more aspect values; responsiveto the at least one of the plurality of aspect probabilities exceeding aminimum threshold, adding metadata to the input image, the metadataincluding a corresponding aspect value of the plurality of aspect valueshaving the at least one of the purality of aspect probabilities; andreranking the search results based on the aspect values indicated by theoutputs of the machine learning system that were missing from themetadata.
 7. The computer of claim 6, wherein, despite the input imagehaving no metadata characterizing at least one of the plurality ofaspect values, at least one of the plurality of aspect probabilities inthe parallel outputs is directed to the at least one of the plurality ofaspect values.
 8. The computer of claim 6, further comprising: adding apublication to the publication corpus, the publication including imageand the metadata.
 9. The computer of claim 6, wherein the paralleloutputs include output location data with a location within an image ofthe plurality of images that has an aspect value of the plurality ofaspect values.
 10. A non-transitory computer readable storage mediumcomprising instructions that when executed configure hardware processingcircuitry to perform operations comprising: receiving, by anetwork-based marketplace, a search query for an item listing of thenetwork-based marketplace, the search query including an input image,the one or more aspect values comprising a category of an item depictedin the input image and/or a color of the item depicted in the inputimage; determining that the input image is missing metadata of one ormore aspect values characterizing, the input image: in response to thedetermining that the input image is missing metadata, generating aninput semantic vector representing the input image; generating a set ofranked search results in response to the search query, the searchresults comprising a plurality of images; providing the input semanticvector as input to a machine learning system of a publication corpus;identifying, by the machine learning system, closest matches between theinput semantic vector representing the input image and image vectors ofpublications images of the publication corpus, wherein the closestmatches are identified based on an exclusive-or operation providing ameasure of similarity between the input semantic vector and imagevectors of publications images; responsive to said identifying,receiving, as a plurality of parallel outputs from the machine learningsystem, a plurality of aspect probabilities indicating the one or moreaspect values that characterize the input image, the plurality of aspectprobabilities identifying probabilities that the input image has acorresponding aspect value of the one or more aspect values; responsiveto the at least one of the plurality of aspect probabilities of theinput image exceeding a minimum threshold, adding metadata to the inputimage, the metadata including a corresponding aspect value of theplurality of aspect values having the at least one of the plurality ofaspect probabilities; and reranking the search results based on theaspect values indicated by the outputs of the machine learning systemthat were missing from the metadata.
 11. The non-transitory computerreadable storage medium of claim 10, wherein the machine learning systemis configured to respond to missing annotation of the plurality ofaspects in the publication images, by forbidding an update of themachine learning system for the plurality of aspects with the missingannotation.
 12. The non-transitory computer readable storage medium ofclaim 11, wherein the machine learning system is configured to respondto missing annotation of the plurality of aspects in the publicationimages by providing the missing annotations of the plurality of aspectsin the publication images.
 13. The non-transitory computer readablestorage medium of claim 10, wherein the machine learning system handlesunbalanced distribution of aspect values of the plurality of aspects inthe publication images, by inversely weighting the aspect valuesaccording to how commonly the aspect values appear in the publicationimages.